Backport

This PR is auto-generated from #16217 to be assessed for backporting due to the inclusion of the label backport/1.5.x.

The below text is copied from the body of the original PR.

Fixes #16211

Nomad servers can advertise independent IP addresses for serf and rpc. Somewhat unexpectedly, the serf address is also used for both Serf and server-to-server RPC communication (including Raft RPC). The address advertised for rpc is only used for client-to-server RPC. This split was introduced intentionally in Nomad 0.8.

When clients are using Consul discovery for connecting to servers, they get an initial discovery set from Consul and use the correct rpc tag in Consul to get a list of adddresses for servers. The client then makes a Status.Peers RPC to get the list of those servers that are raft peers. But this endpoint is shared between servers and clients, and provides the address used for Raft.

Most of the time this is harmless because servers will bind on 0.0.0.0 anyways., But in topologies where servers are on a private network and clients are on separate subnets (or even public subnets), clients will make initial contact with the server to get the list of peers but then populate their local server set with unreachable addresses.

Cluster administrators can work around this problem by using server_join with specific IP addresses (or DNS names), because the Node.UpdateStatus endpoint returns the correct set of RPC addresses when updating the node. So once a client has registered, it will get the correct set of RPC addresses.

This changeset updates the client logic to query Status.Members instead of Status.Peers, and then extract the correctly advertised address and port from the response body.

Notes:

• The end-to-end testing required for this means it's going to just miss the 1.5.0 GA. This is probably safe to land in a patch release without worrying about backwards compatibility, because anyone with this topology is already using the server_join workaround described above.
• This also closes fix wrong control plane discovery, when serf advertised address located in network, unreachable from clients #11895, which unfortunately got abandoned because we had some trouble understanding exactly what the problem was.
• I've done some bench testing of this with a local development cluster, some extra IP addresses, and iptables rules.
• I also stood it up on a real multi-host cluster using the changes in our E2E config from E2E: add multi-home networking to test infrastructure #16218 and everything seems to work as usual.
• This is outside the code path of the non-Consul cloud discovery features, but I also tested it by disabling server join via Consul (by just breaking the consul config block) and used AWS EC2 auto-join via tags and that works just fine as well.